Method and system for real-time virtual 3d reconstruction of a live scene, and computer-readable media

ABSTRACT

A method, system and computer-program product for real-time virtual 3D reconstruction of a live scene in an animation system. The method comprises receiving 3D positional tracking data for a detected live scene by the processor, determining an event by analyzing the 3D positional tracking data by the processor, comprising steps of determining event characteristics from the 3D positional tracking data, receiving pre-defined event characteristics, determining an event probability by comparing the event characteristics to the pre-defined event characteristics, and selecting an event assigned to the event probability, determining a 3D animation data set from a plurality of 3D animation data sets assigned to the selected event and stored in the data base by the processor, and providing the 3D animation data set to the output device,

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/875,826, filed Jan. 19, 2018 (published as U.S. Patent PublicationNo. 2018-0211432), which is pending; which is a continuation of U.S.patent application Ser. No. 14/410,943, filed Dec. 23, 2014 (now U.S.Pat. No. 9,886,788); which is a U.S. National Phase Application under 35U.S.C. § 371 of International Patent Application No. PCT/EP2013/064152(published as WO2014/006143), filed. Jul. 4, 2013; which claims thebenefit of priority under 35 U.S.C. Sec. 119 of European Application No.12174999.8 filed Jul. 4, 2012; all of which are incorporated byreference in their entireties.

The invention relates to a method and a system for real-time virtual. 3Dreconstruction of a live scene.

BACKGROUND

Conventional live broadcasts are more or less static, having only weakways for interaction with the consumer. The broadcasted live videofootage cannot be influenced by the consumer. Users are interested inhaving individual access to live imagery. Therefore, the live scenes arecloned or to be better said reconstructed in a virtual environment. As aresult the consumer can have individual interactive access to virtuallive scenes.

In general the approach is applicable to numerous areas. Beginning withindustrial workflows or in sports broadcasts to interact with thevirtual visualization of a live reconstruction of real motions fromcharacters and objects. As example, reference is often made to abroadcasted football match.

Document EP 1 796 047 A1 refers to a procedure for some simpleanimation. It's not sufficient and designed for the requirements of alive environment and not for an automatic reconstruction of realisticcharacter animations.

SUMMARY

It is an object to provide an improved technology for real-time virtual3D reconstruction of a live scene.

A method for real-time virtual 3D reconstruction of a live scene in ananimation system is provided. The animation system is having aprocessor, a data base, an input device, and an output device. In themethod, 3D positional tracking data for a detected live scene arereceived by the processor. Different systems and methods for detecting3D positional data are available. In a simple case, one positionalcoordinate is provided for objects of the live scene observed by thepositional data tracking system. In the method proposed, the 3Dpositional tracking data are analyzed by the processor for determiningan event. Such step may also be referred to as event detection. In theprocessor, the event detection comprises determining eventcharacteristics from the 3D positional tracking data, receivingpre-defined event characteristics, and determining an event probabilityby comparing the event characteristics to the pre-defined eventcharacteristics. Following, an event assigned to the event probabilityis selected. A 3D animation data set is determined from a plurality of3D animation data sets assigned to the selected event and stored in thedata base by the processor. The event detection using event probabilityas a decision criterion may also be referred to as early event detectioncompared to normal event detection. The 3D animation data set isprovided to the output device connected to the processor. Through theoutput device the 3D animation data set may be provided to a displaydevice by wired and/or wireless data communication.

The processor may be implemented in a server device connectable to oneor more display devices.

The method proposed allows a real-time presentation of the 3D animationdata on a display device, namely a time parallel fashion with respect tothe detection of the 3D positional tracking data by the tracking systemused.

The event detection may be done by only analyzing positional track data.

An animation data queue may generated by several times repeating one ormore of the steps of receiving 3D positional tracking data, determiningan event, determining a 3D animation data set, and providing the 3Danimation data set.

An event correction may be performed for the animation data queue by atleast one of deleting one of the 3D animation data sets from theanimation data queue and modifying one of the 3D animation data setsfrom the animation data queue.

The step of adjusting one of the 3D animation data sets may comprisesubstituting a 3D animation data set selected from the animation dataqueue by a new 3D animation data set determined to closer animate theselected event.

The 3D animation data set may be provided as a plurality of 3D sub-datasets being provided with 3D animation data of different data formats,the different data formats being adjusted for different display devices.In case of having the animation data queue generated, different queuesmay be provided, each queue comprising 3D animation data of differentdata formats.

The plurality of 3D animation data sets stored in the data base maycomprise simulated 3D animation data and/or pre-recorded motion capturedata. The simulated 3D animation data may represent an object based on aphysical model.

DESCRIPTION OF FURTHER EMBODIMENTS

Following, further embodiments will be described, by way of example,with reference to figures. In the figures show:

FIG. 1 a schematic representation of a workflow of incoming positionaldata which is analysed in the early event detection, with the animationdatabase lookup the best matching animation is found and is insertedinto an animation data queue to be rendered,

FIG. 2 a schematic representation of an optical 3D tracking systemconfigured to provide positional data with respect to a live scene“football match”,

FIG. 3 a schematic representation of top view on hip position over time,

FIG. 4 a schematic representation for different time notations,

FIG. 5 a schematic representation of a football pitch on which zones fora team are indicated playing towards the right goal,

FIG. 6 a schematic representation of an action radius and an extendedradius for a player,

FIG. 7 a schematic block diagram for a process of animation ofcharacters,

FIG. 8 a schematic representation of an animation data queue with threestages,

FIG. 9 a schematic block diagram for an early event detection,

FIG. 10 a schematic representation of live scenes of a football matchwith respect to early event detection,

FIG. 11 a schematic representation of the live scenes in FIG. 10 withrespect to early event detection with event correction,

FIG. 12 a schematic block diagram for lookup in an animation database,

FIG. 13 a schematic representation of reconstruction stages of a 3Danimation data set placed in the animation data queue,

FIG. 14 a schematic representation of a virtual 3D reconstructionsystem,

FIG. 15 a schematic representation of a double screen system showing thelive scene and the virtual 3D reconstructed animation, respectively, and

FIG. 16 a schematic representation of a top view of a reconstruction,

By sing the input data (positional tracking data) of a real-timetracking system a virtual 3D reconstruction of a live scene may beprovided with a short delay. This can result in a realisticvisualization of the live scene. It can be individually andinteractively controlled—used e.g. for sports analysis of a broadcastedsports event. A reconstruction of a live football match could beinteractively viewed, camera and analytical elements individuallycontrolled. The consumer can influence the camera, the visualization andits procedure.

The reconstruction method may use pre-recorded motion-capture animationswhich may be stored in an animation database along with attributes asthe animation's speed and an assigned category (e.g. shot or tackling).

FIG. 1 illustrates the workflow of incoming positional data which isanalysed in the early event detection, with the animation databaselookup the best matching animation is found and is inserted into theanimation queue to be rendered. As the motion-capture animation is notmatching the real sports motion in detail using physical behaviours andoptimization helps to revise and reapply these specific nuances anddetail. The result is a computer graphical visualization of theanimation. E.g. it can be used to be embedded m a fully virtualenvironment or as an overlay to a recorded or live video stream.

Referring to FIG. 1, after starting the procedure in step 100,positional tracking data for a detected live scene are received by aprocessor of the virtual reconstruction system in step 101. Following,in step 102 an early event detection is performed. As the result of theearly event detection 102, an animation data set is provided (step 103).According to the embodiment schematically depicted in FIG. 1, in step104 the animation data set is corrected providing a corrected animationdata set (step 105). Such correction may comprise at least one ofanimation blending and animation mixing which is described in detailbelow. Further optimization may be performed in step 106. The animationdata set provided by the process is visualized in step 107. In step 108it is checked whether more animation data shall be outputted. If not,the procedure will end in step 109.

Current techniques on tracking systems (used in sports broadcasting)provide not enough quality for a direct realistic character animation(e.g. with key-frame animation and IK). These systems are providingnormally only one positional coordinate for each character (athlete) orobject (ball). The solution to this is in having an animation databasealong with pre-recorded motion-capture animations. Positional data andevent detection are used to recognize patterns and find the most likelymatching animation. A realistic flawless aesthetic appearing motion is ahigh priority for the method. In the same way the use of motion captureis also its disadvantage as pre-recorded data varies in certain detailsfrom the live scene, it is always an abstraction. To solve this, theincoming data needs to provide additional data for the stage of databasepattern matching and later on in the optimization.

The quality of the incoming data can be sparse. Due to the pre-recordedmotion capture animations the reconstruction will always look convincingin terms of a realistic motion. A great advantage of the method is itsflexibility. It works with very different tracking systems as e.g. forfootball players a two dimensional (x, y) coordinate representing theposition of the player on the football pitch and for a ball threedimensional (x, y, height), its timestamp and a unique id (identifyingthe player) are needed. Also the time interval at which input data comesin for a certain player can vary. In the end there can be a number ofdifferent methods to generate the input data, from manual creation ordrawing running traces to video-based or chip-based tracking systems asthey are currently used at a growing number of sport events.

High-End tracking systems can provide multiple positions for a singlecharacter (e.g. for every hand and foot) these can be used to optimizethe reconstruction. Besides the input data, the physics system andlogical parameters are also influencing the optimization state. E.g. afootball player's head is oriented towards the ball.

The generation of the reconstruction must happen live—only with a shortdelay to the happening of the live scenes. The method targets a delay ofmuch less than one second, regarding the duration from receiving thepositional input data until the current time of animation playback.There are two stages in the workflow which are critical to the delay,namely the event detection and the animation detection. The first issolved with an early event detection, which makes assumptions andprobabilities based on the scenery. The latter is solved using aprobability list of animations which are inserted into the animationqueue and later on can he modified or replaced based on new detailsdetected afterwards in the early event detection and/or changes in theprobability list of animations.

Target devices for this second screen experience are personal computers,tablets, smartphones and the television device itself, as mostup-to-date device allow downloading and processing of apps. A system hadto be developed to allow the reconstructions on each device whileregarding its limitations. The proposed system provides differentanimation sets, for some target devices the number of animations wille.g. be 50 as on high-performance devices there could be more than 1000.Of course influencing download size and computational performance. Theanimation queue for each device is calculated on a server and only theresulting animation queue data (which is small) is used on the devices.

For the 3D reconstruction to work positional data is needed as an input.So this input data is given or in other words delivered to the method.The input data should minimally be a single position (center of mass) ofa characters I objects in the absolute three-dimensional Cartesiancoordinate system. In most situations a two-dimensional position forcharacters is already sufficient e.g. in football. In the technicallysimplest case one positional coordinate is provided per character andobject. The system then uses a lot of assumptions, e.g. for a footballthe position is given with 25 frames per second, the system now assumesthe rotation by applying physical models. E.g. for a football player forits given position the complete pose is an assumption—the system usesspeed of the players movement to find a matching animation.

When wanting a live reconstruction an automatic or semi-automatictracking system may be used. The tracking system may at least deliverpositional data multiple times per second with a short delay. Naturallywhen having only one coordinate per character the animation will differin a lot of details to the actual occurring. More coordinates for keyjoints (e.g. feet, hands) of a character are making the quality and thecloseness to the real motion radically better.

There are different types of tracking systems present on the marketwhich match the requirements to deliver positional information inreal-time. Other techniques as known from motion-capture systems are notin a direct competition to these systems but could evolve into thatdirection and can potentially be used to deliver the input data.Tracking systems are currently used at a growing number of sport events.

After an introduction of the common types of tracking data systems theseare generally referred to “tracking system”. FIG. 2 illustrates anexample for an optical tracking system where six cameras 200 are placedaround a pitch 201 to detect the player's position on the pitch 201 as a2D coordinate.

As an example, chip (e.g. RFID) based tracking systems are available,such as inmotiotec from Abatec Group AG or RedFIR from Fraunhofer IIS.Chips are attached to the athletes or placed within the ball. Also,optical tracking systems are known. Camera (e.g. stereoscopic cameras)based systems are successfully installed on the broadcast market. Usedfor large events as FIFA World Cup and UEF A Euro, but also in footballleagues. Manufacturer is e.g. Tracab.

There are other systems which could potentially be used, however thesedeliver data which is so detailed that a direct animation could workwell enough, making the system redundant. But e.g. Microsoft's Kinecthas the disadvantage of working only for a small number of characters.For motion-capture systems such as from Vicon or XSens, actors have tobe equipped with a special suit. Both approaches make it impossible tobe used for sports broadcast. But future development might lead into thedirection.

There are numerous methods for generating the input data. Besides theautomatic or semiautomatic generation from a tracking system, there arealso manual approaches. E.g. setting a position at certain key-frameswhile interpolating positions in between. Or e.g. drawing running traceson a 2D top view of the field. While the system proposed may be workingwith these approaches a live reconstruction is in this case notpossible, as the delay would be much larger.

A configuration is used to inform about certain things important forinterpreting the input data. Coordinates of key points as the origin ofthe coordinate system and e.g. for a football pitch corners, mid-pointand position of goals. Also the Ids in the positional data used toidentify a position belonging to a certain character or object asreferees, players, ball, etc.

A tracking system delivers positional information in real-time, whichmeans with only a short delay and, for example, with a rate of 25 timesper second. E.g. for a football player a two dimensional (x, y)coordinate may be provided, which defines the position on the footballpitch; for a ball the height should be given additionally. The positionmay represent the centre of mass I mid or hip position verticallyprojected onto the football pitch. The accuracy may be less than +/−0.5meter, and positional coordinates may be represented in an absolutecoordinate system.

The time interval at which input data comes in for a certain characteror object can vary, so basically it is not important if a characterposition once every two seconds or 25 times a second. However for thesystem to work live with only a short delay a position for everycharacter or object position may be provided at least five times asecond.

For a football match the tracking system can deliver positions for allplayers, balls and referees along with an id to uniquely identify these.A tracking system may deliver at least one position per object, but itis also possible delivering multiple positions e.g. for legs, arms,hands, head etc. The more positions are captured the more exact is theevent data of the tracking system resulting in a more detailedreconstruction.

If the tracking system delivers more positions per character these maybe used in two ways. At first, it may be used for a more detailedanimation database lookup to insert the best matching into the animationqueue. Secondly, it may be used in addition or as an alternative for thelater optimization, e.g. in football the hands for field players arecorrected afterwards as the feet are more important.

From the positional data events may be determined which may also bereferred to as event detection (see step 102 in FIG. 1). E.g. if anathlete is jumping or shooting a ball by observing the positions anevent can be identified. By using logical rules e.g. if a ball movesquickly away from a player an event “shot” can be detected. Furtheranalysis could lead into an event “pass” if it is received by anotherplayer or if received by a player of the other team it can be an event“failed pass”.

Conventional event detection known as such will lead into a delay ase.g. the failed pass could not be detected at the moment of the shootingthe ball, but only a few moments later when the opposing player gets theball. Event detection may be used for statistics like running distanceof a player over the full match, counting of ball contacts or passeswhere the delay is minor.

As the early event detection may use just positional data as input it iscompatible to any tracking system that delivers such data. There are noconventions on which events are detected, tracking systems handle eventdetection differently and each with its own delay.

The animation techniques used may be as such state of art. Thesetechniques are used extensively and are adopted to match the key focusof delivering a fast reconstruction with good quality. In the following,these known techniques are described on a very basic level.

A categorization into characters and objects may be done which is onlyexemplary to describe the workflow. Techniques used by either of thesecan also be useful for the other and can be mixed. Both terms are usedhereafter whenever either category makes more sense for descriptionpurposes. Objects as a ball of a football match. Has no complexanimation by itself, so motion-capture doesn't make sense, but aphysical simulation. If a ball is falling to the ground it bounces upagain and rotates. This slight behaviour is hard to be correctlydelivered by tracking systems. As the convincing animation is thehighest priority it is simulated by a physical model, rather than beingas close as possible to the live reference. This ensures the convincingmovement of ball—in the same manner as for characters this is anabstraction of the live scene. A character has human like or relatedattributes as several joints as elbows and knees. The animation iscomplex. Pace, motion and physical behavior are aspects which cannot besimulated easily. To overcome this situation pre-recorded animations areplaced optimally in an animation queue. But in certain cases also here aphysical simulation can be useful as described in a physical behaviour.The character workflow is more complex and is described in the followingchapters.

Prior to the real-time virtual 30 reconstruction of a live scene,motion-capture animations are recorded in a studio or on the footballpitch, captured by the motion-capture systems. Motioncapturing is atechnology known as such in different configurations. E.g. Vicon andXSens are vendors of such systems. The technology is further describedin Menache, Understanding Motion Capture for Computer Animation, (2ndEdition) from Alberto published by Morgan Kaufmann (ISBN:978-0123814968, 2010) and Kitagawa et al., MoCap for Artists: Workflowand Techniques for Motion Capture, published by Butterworth Heinemann(ISBN: 978-024081000 3, 2008).

8 illustrates an animation queue where animation blending and mixing areused, e.g. in Stage 1, a ‘Walking’ animation 804 is blended into‘Running’ 806. And a ‘30% Walking’ animation 805 is mixed with ‘70%Running’ 808.

Animation data sets representing the animations to be outputted areplaced in order in an animation data queue. Each animation representedby an animation data set starts at a certain time and ends at a certaintime. During a live animation only two may be active at a given moment.It is possible to smoothly blend between two animations and also to mixtwo animations.

The animations placed in the animation data queue may be amended. Onekind of such amendment may be referred to as animation blending which isto change the weight of two animations to blend the active animationover into the other. Such process may also be referred as cross-fading.The length of the transition is relevant.

Throughout this short time range both animations have to be as similaras possible to proceed a smooth blending. The information are stored asa transition in the animation database, holding all matching posesbetween two animations that are good enough to proceed a well lookingblending.

Another process for amending the animation data queue may be referred toas animation mixing. Rather than blending between two animations in thistechnique these are mixed over a period of time. One jumping whilerunning animation and a running animation mixed can lead into a lowerjumping while running animation. Animation mixing overcomes animationblending in terms of complexity. In animation blending a single timerange is used to blend over. In animation mixing a transition graph isgenerated resulting in a path which is the best way to blend into theother animation for every frame. So at each time the animation can beblended over, with the effect that these can be mixed fluently.

A locomotion process may be used for overcoming problems of animationblending and/or animation mixing. Sometimes feet are sliding whileblending or mixing. This happens as both poses are slightly differentover the blending time range while both animations might fit well atexactly in one frame in their poses. Animation method for changinganimations e.g. a running animation the step length ischanged—shortened/widened. The motion itself stays intact but isadjusted to fit another distance or match another speed. This solvesproblems as that the feet are not sliding. Another field of locomotionis e.g. to walk hill upwards or walking a curve where the footsteps haveto move upwards or sideward at each step. There are a number ofpublications to this, Park et al., On-line Locomotion Generation Basedon Motion Blending, published by ACM in SCA, 02 Proceedings of the 2002ACM SIGGRAPH/Eurographics symposium on Computer animation (p. 105-111,2002. ISBN: 1-58113-5734); and Johansen, Automated Semi-ProceduralAnimation for Character Locomotion (published as Master Thesis atDepartmentof Information and Media Studies at Aarhus University, 2009).

During an optimization stage kinematics may be used, Inverse kinematicsare imaginable to a Marionette like system, when setting the position ofa hand, arm and elbow will follow. Furthermore Human IK is a specificsystem for humanoid characters where in this example the hand of anupright standing character could be set to the ground—the body of thecharacter will bow physically correct to match that hand position. Thisis used to try matching a pose or e.g. feet.

A physical model may be used for providing certain input parameters to amodel, as distinctive positional coordinates and speed for a ball. Withthe physical model a trajectory is rendered to provide the curve theball travels and its rotation.

A physical behaviour refers to a more complex logical behaviour and maytake into account the complex behaviour of humans in terms of body partsand logic. An example is the collision of two characters. The collisionmay happen under unique circumstances as players run at a differentspeed and direction each time and therefor tackle each other at otherbody parts with another force. It is impossible to cover all differentvariations of the collision with motion-capture so the tackling needs tobe simulated. Input parameters could be a position and a force which isexerted. E.g. on the upper body, which lets the character fall down in arealistic way. The animation for this is completely calculated in arealistic way. It is possible however to provide a normal animation atwhich the physical behaviour outgoing and takes responsibility for thefollowing behaviour. In some cases the physical behaviour makes senseand is preferred over the standard motion-capture animation forcharacter e.g. if these tackle each other in a unique way where notmotion-capture animation has been recorded before. However the physicalsimulation only can be applied in special cases where a physicalinteraction and forces are applied, e.g. a standard walking animationcurrently can't be done with a physical simulation. A high-end system ise.g. euphoria from NaturalMotion.

An extreme example is for american football when all players are jumpingonto each other trying to get the ball, this is hardly achievable to becovered with motion capturing and targeting a reconstruction later—asthe behaviour of the players is always unique (players are at differentpositions etc.).

The animation database may be created upfront for a certain discipline,such as football. It is created a long term (months) before the actualgeneration of the real-time animation, as it takes time to integrateanimations and logic for that discipline into the system—foremostanimations have to be recorded with motion-capture and integrated intothe database.

The sport discipline to be reconstructed has to be analyzed. A list ofunique motions which are executed by the athletes has to be created.Often these are more than a thousand animations e.g. a hundred differentvariations of a shot. An actor (football player) is executing themotions from the list while ensuring that their appeal is as similar aspossible to the motions that most athletes will execute. Additionallythere are animations recorded with two or three actors at once and withthe ball being involved as tackling or shots. The individual motions foreach actor/ball are separated in the animation database but itsrelationship is stored and can be used to recreate the interactionbetween athletes/ball in the animation queue for playback.

Each animation has a length of normally less than a second or a fewseconds. E.g. a running loop, a trick shot with the left foot etc.Animation is mirrored automatically which doubles the amount ofanimations in the database. Most animations are a bit longer ahead andbehind than the actual action (e.g. shot)—to be used for blending itover into other animations.

A motion-capture animation can be split into segments itself e.g. to usethe animation of the upper and lower body separately, which is alsostored in the database. In some cases this can be useful to have anoptimized blending or better fitting reconstruction.

The animation database lookup will be more accurate the more detailedthe input pattern is. In simple terms an input can be a certain action,like a shot or e.g. a combination of positions of a hip and joints likethe left foot. During the lookup the animation database is parsed usingthe input parameters. The actual lookup detects motion patterns overmultiple frames and not single poses at a frame. The best matchinganimation will be inserted into the animation queue. The result is areconstruction where the virtual character's motions match those of thereal athlete. E.g. while running the left foot will be in front of thecharacter (in the reconstruction) whenever it is in the live scene. Anexemplary animation database lookup is illustrated in FIG. 12.

To enable a fast animation database lookup, attributes may be definedfor each animation. Basically attributes are metadata which distinct theanimations from each other e.g. the information if the animationincludes a ball. For a shot this would be true and for a runninganimation it wouldn't.

Logical attributes may be partially generated automatically, but someinformation is may be provided manually by an operator. Some examplesare: Category of the animation (running, shot, tackling . . . ) withsubcategories e.g. for shot (pass, goal shot.). Additional informatione.g. for a shot with which foot it is executed, furthermore if with theside foot, a full instep kick, etc. If the animation is loopable—true ifthe animation can be restarted from beginning after it ends.

Probability—how often will the animation probably be placed in theanimation queue. E.g. a running animation is used very often, anoverhead kick only rarely.

Technical attributes in general may be automatically apprehendedattributes retrieved by observing the animation files. Some examplesare: (i) Average speed—speed of different phases e.g. beginning phase(0%-20% of length), ending phase (80%-100% of length) and in the middle;(ii) spacial trajectory which the hip of a character moves along.

If animation interacts with object (ball) or character (teammate oropponent). Some attributes can be adapted to better match the input ofthe lookup. For example the speed of animation can be modified—if theanimation has a length of one second it can be played with double speed,so the motion is executed faster and then has a length of 0.5 seconds.

Movement patterns for player and ball like their trajectories (e.g.special movement of hip of the player over the length of an animation)can be compared to the incoming positional data. If logical andtechnical attributes basically match and deliver four usable animations(like in FIG. 12) these patterns help to refine the probabilities foreach of those animations. FIG. 3 illustrates a trajectory of hippositions (curved line) over time of an exemplary animation from a topview (XZ-axes). FIG. 3 shows the path of an animation from top view.This path can be rotated and its speed adapted. Points A, B and C can bepart of the pattern lookup and used for the first lookup if these areadequate the trajectory is checked in further detail. A is a position ofthe hip in first frame of the animation. B is a position in middleframe. C is a position in last frame. If the input data includes a ballclose to the position of the player or the player is interacting withthe ball, the pattern lookup searches primarily for animations whichinclude a ball, the trajectory within the positional input data and theanimation from the database are compared and therefore influence theprobability/result.

An additional pattern approach is to use usage details. Whenever ananimation is placed in the animation queue for playback this usageinformation will be inserted into the database. E.g. a count of howoften the animation has been inserted into an animation queue, thesummed duration of usage, under which circumstances which detectedevents or input patterns) and also information about the animation whichhas been inserted directly before and after this animation in the queue.The result is a complex graph providing information on how well theanimation fits into the given queue under the current circumstances.

Information to blend and mix animations may be automatically generatedthrough finding matching poses between two animations, and are in somecases optimized manually. These are stored as transitions and transitiongraphs. This enables blending at certain frames and mixing of animationsat any time.

Following, further aspects of the virtual 3D reconstruction technologyare described.

Position, timestamp and id of the players are received at intervals fromthe tracking system and used to generate a 3D reconstruction of theathletes' motions and objects as the football—resulting in a computergraphical visualization. For a motion-capture animation which has alength of a second the delay normally would be a second. A second ofpositional data has to be received and only then the decision could bemade if it is the best matching animation.

There are situations where this is not a problem e.g. a broadcast-signalfrom a live event is normally also a few seconds behind the live eventuntil it appears on the television screen of the consumer. Sotheoretically the reconstruction could still be in sync with thebroadcast, but in certain cases the delay has to be as short aspossible, as there are steps to make the visualization available toconsumers etc. which also need some time.

To enable a workflow which is as fast as possible—early event detectionis used—the generated probability tree and probability animation listretrieved from the animation database lookup are the key elements inthis solution.

The FIG. 4 illustrates the different time notations that are used inthis process: between playback time and live time is a delayed timerange and beginning at live time into the future the prediction timerange. In FIG. 4, A represents an offense player running towards goaland B represents a goalkeeper of the other team, defending goal. FIG. 4shows changes of early event detection over three exemplary live times,given live time is the moment or time at the live time (current time oflive event). Before the playback time is the past, animations havealready been placed in the animation data queue and cannot be changed.The playback time is the current time of playback in the animation dataqueue. Any animation or animation weight after this time can be changed,which is done by the early event detection, and used as forenm. Actionswhich have been detected in the delayed time range did already happen,these have a high influence (e.g. 70%), whereas actions predicted withinthe prediction time range have a low influence (e.g. 30%) in the earlyevent detection. But the influence values vary, depending on thesituation and actions (dynamic attributes). The delayed time range isnormally small (e.g. 0.1 s) and the prediction time range longer (e.g. 1s). Early event detection, event correction and the placing in theanimation queue may be processed in an interval of at least about 25 to50 times per second.

In the case of football two ways for animating the ball object may beconsidered. First is using the trajectory from motion capturing theother to use a physical model to simulate the behaviour of the ball.Both can be mixed, depending on the situation.

A physical model may be used for animating the object in a realisticway, it just needs a minimum amount of input parameters e.g. certain keypositions, speed and acceleration. Ball curve, bouncing or collisionwith other objects are calculated by the model. E.g. for the ball anassumption of the ball's flying curve will be simulated, which mightdiffer from the coordinates and timing of the positional input data.When the ball is high in the air flying a curve this is normally not aproblem as long as the start and landing point of the ball match withreality, so these key points are set as fixed for the physical model.

If the ball trajectory comes from motion capturing it also differs fromreality. E.g. if a player is dribbling, the positional input from thetracking system might not be used at all, as a perfect animation mightnot be found in the database. From this input data just the informationthat the player is in ball possession and dribbling is used, while therealistic animation of the ball comes partly from the motion-captureanimation itself (where the ball was recorded along with the player) andcan be partly simulated and corrected by the physical model.

Another role of the physical model is to calculate an assumption of theball trajectory e.g. right at the beginning of a shot. The assumption ofthe trajectory can be used in the early event detection for the dynamicparameters (where the position of the ball is used). So it has influenceon the probability tree of the characters,

Early event detection is used which may be implemented with a so-calledprobability tree. The focus of the probability tree is to always haveup-to-date information on possible trends, to know as early as possibleif the chance for an upcoming shot arrives or into which direction itgoes. To solve this, early event detection works with assumptions andprobabilities. Based on the current situation certain attributes(dynamic attributes) are apprehended these together with attributes thatare retrieved before a match for every player (fixed attributes) areused to guess how the player will probably act next. This approach canbe implemented with a state machine for each player, AI uses eachplayer/ball and their states to better predict their actions. An actionis for example a shot on the goal, a dribbling or a tackling. Apart fromthat early event detection is also about simple logic like standing,walking or running, this is a standard behaviour for which no complexpredictions have to be made. E.g. the current running speed anddirection and predictions for these are used as an input for the earlyevent detection and therefor do have an influence on the players nextactions.

Fixed attributes are set before a match. Some examples are: Generalposition of player (offense, defence, goalkeeper . . . ); and abilitiesof player (skills for shooting, passing, tackling, dribbling . . . ).Dynamic attributes are dynamically apprehended at current moment of thelive scene. Examples are: Current position of opponent players andteammates (chance for pass); and the player having free view (goal shotpossibility) at goal, which can influence a possible shot on the goal.

In which zone of the field the player is highly influences itsbehaviour. In front of the opponent's goal the player's offense actingis more crucial, dribbling gets more difficult as defenders are moreattentive, so the opponent might tackle etc. In the neutral zone in themidfield the actions would be more considerate. A simple zoneclassification is illustrated in FIG. 5. Action radius and extendedaction radius may be used. If ball is in action radius (close) orextended radius (pass possibility) is important for the eventual event.Passes are more likely the closer a teammate is as illustrated in FIG.6. Condition, updated attributes of the players abilities (fixedattributes), which can change over the runtime of the match, e.g.endurance level will decrease if he is running a lot.

Changes of dynamic attributes over time (e.g. 1 second range before livetime), e.g. if for the goalkeeper ‘caught ball’ has been detected (iswithin 1 s range), probabilities for him securing the ball will be high,followed by a throwing of the ball towards a teammate.

Predictions can be influenced based on other players actual actions andprobabilities, e.g. if it probability for a forwarder for making a shoton goal is high, the goalkeepers probability for trying to catch theball will increase.

Action radius and extended action radius are used. Whether the ball isin action radius (close) or extended radius (pass possibility) isimportant for the eventual event. Passes are more likely the closer ateammate is.

FIG. 7 shows a schematic block diagram for a process of animation ofcharacters.

After starting the process in step 700, information about the eventdetermined is received in step 701. According to step 702, an animationdatabase is looked up for providing, according to step 703, a basicanimation represented by a basic animation dataset. In step 704 it ischecked whether the animation represented by the basic animation dataset needs correction. If it is determined that correction is needed instep 705 correction of the basic 3D animation data set is done. Theevent correction is further described below. If, according step 706, theselected 3D animation data set is sufficient, at least one of actionanimation and physical behaviour is used in step 707. In case of use ofaction animation the animation database is looked up and actionanimation is received according to steps 708, 709. Physical behavior maybe used according to step 710, finally providing optimized animationdata in step 711. The process is ended in step 712.

The probabilities can be influenced by a pattern recognition whichobserves patterns m situation and (tactical) movement of players.

FIG. 8 illustrates an exemplary animation queue of a character. Stage 1is a basic animation, which is a blending of two animations. Stage 2 isa here action animation and physical behavior basic animation. Stage 3is an optimization stage that includes correction of details, has itsown weight, and can he only on certain body parts (Here-Foot). The livetime is current time of the live event, e.g. a football match 802. Thedelayed time is the current time of animation playback (virtual 3Dreconstruction) 801. Depicted left to the delayed time is thepast—animations in the queue which already have been played. Betweendelayed time and live time is a small buffer, the above described delay.Right to the live time is the prediction time range 803, which isgenerated based on the results of probability tree and list, and is aprediction of the future events of the player. There are three stages inthe animation data queue. Stage 1 809 is the basic animation e.g.general idle, walking and running animations. Stage 2 812 is the actionanimation combined with the physical behaviour. Stage 3 814 is theoptimization stage. All stages can be mixed to certain amounts and alsoin a way that only affect parts of the player's body e.g. anoptimization stage 814 might only correct the left foot to match a balltrajectory 813—which is an addition to the full body animation resultingfrom Stage 1 and 2. In FIG. 8 Stage 1 and Stage 2 have a summed weightof 100% at each time—the weight is illustrated by the height of eachanimation, a 100% is filling out the full height as in ‘Walking’ ‘804 ofa layer (Stage 1 consists of two layers) while a weight of 30% (as forthe ‘30% Walking’ animation 805 fills out 30% of the layer's height.There are often cases where an animation of a layer is blended out andanother is blended in—e.g. at delayed time where ‘Running’ 806 starts tofade out and ‘Action Animation Goal Shot’ 810 starts to fade in. The twoanimations of Stage 2 overlay a ‘0% Running’ 807 animation of Stage1—this allows to have no influence of the ‘Running’ animation while the‘Action Animation Goal Shot’ and ‘Physical Behaviour Collision withOpponent’ 811 is played, Below the stages there are three descriptionsof remarkable time stamps. The left mark 815 indicates the moment of thegoal shot as it has been detected by the event detection—this moment hasto match the moment of the shot in the animation ‘Action Animation GoalShot’. A second line mark 816 indicates the adjusted moment where the‘0% Running’ animation of Stage 1 is adjusted to the moment where the‘Action Animation Goal Shot’ is faded out to make the blending work. Athird line 817 marks the moment where ‘Physical Behaviour Collision withOpponent’ fades out and a mix of ‘30% Walking’ and ‘70% Running’ 805,808 fades in.

FIG. 9 shows a section of the early event detection. It starts withreceiving the information from early event detection or event correction901, 902 from the last iteration or preceding part of the eventalgorithm. In reference 903 it is decided if the player has currentlyball possession. Reference 904 would be the handling without a ball—inthis simplified example of a subordinate branch of a tree the offenseplayer has the ball. There are three leafs for which a probability iscalculated for: dribbling 907, pass 910, goal shot 913. These areretrieved by observing attributes of the current situation. E.g. inreference 905 it is checked if an opponent is within the extended radiusof the player, if so fixed and dynamic attributes are observed inreference 906 the concept is explained above. If the opponent is notwithin the action radius, the player has enough space for dribblingtherefor resulting in a higher probability for dribbling 907. In asimilar way probabilities are calculated for a pass observing thecurrent situation, if it allows a pass to a teammate 908, 909 and 910.

For the condition “Goal Close” 911 the fixed attributes of the positionof a player and its shoot ability are observed 912. In this case it's anoffense player which has a strong shot. If the observed dynamicattributes reveal a free view to the goal and the distance is within theextended radius it results in a high probability that (reference 913)the player will take a shot on the goal. By using patterns forsituations (e.g. general movement of all players, as seen from top-viewor the movement of individual players) reoccurring situations can beidentified and used for enhancing the probabilities.

This is a small excerpt from a full tree, one possible branch—there canbe many more attributes, branches and sub branches (as indicated byreference 914) and a probability like pass can be refined into aprobability for a short/long pass, low/high pass etc.

The probability tree is generated for each player (step 915). Theprobability tree for basic animation is different of those for actionanimation as the number of possible outcomes/animations is smaller forthe basic animation (e.g. walk and running cycles). For the actionanimation the delivered tree is larger as there are more opportunitieswhich the player can probably do (e.g. many variations of shooting,tackling, dribbling).

The early event detection which results in a probability tree.

Following, a further example is referred to.

FIG. 10 illustrates the early event detection from the top-view on afootball pitch. The probabilities for player B change throughout thedifferent moments of the scene. B's chances to receive the ball increaseas the ball comes towards him until finally to 100% when he received theball. When the probability increases over a threshold (e.g. 40%) theanimation is inserted into the animation queue for playback. If theprobability for Shot on Goal grows over the threshold both animationsare mixed. E.g. receive ball animation is played with a weight of 67%(while having 70% probability) and the shot on goal animation is playedwith a weight of 33% (while having 55% probability). Both animations areplayed and the mix weights change during the process of the scene andits probabilities.

There are certain rules to each sport event which have to be regardedand detected, which can influence the dynamic attributes. E.g. the rulesfor football: offside, goal, throw in etc. This makes the configurationof the early event detection specific to a sport.

An event correction observes the probability tree, probability list andthe active animations in the queue. If the observations are resulting ina needed correction to the animation queue, it is adjusted to match thenew circumstances e.g. a new animation has to be inserted. For the casethat an action animation has already been inserted and playback has beenstarted, the mechanism is to correct the animation queue in blending theunneeded animation out and blending the correct one in, or adjusting anexisting one so that it matches the new requirements. The eventcorrection may be part of the workflow as illustrated in FIG. 1. FIG. 7illustrates its place directly after stage 1 (reference 703)—basicanimation and before stage 2 starts (reference 706).

FIG. 11 illustrates a similar scene as in FIG. 10, but in this casegoalkeeper C gets into the way and catches the ball before player B canreceive it. At first the receive ball animation is inserted into theanimation queue but is then aborted when goalkeeper C catches the ball.The receive ball animation's weight is decreasing quickly towards 0% andis overlaid by a normal running animation whose weight increases.

Referring to FIG. 10, three frames 1000, 1001, 1002 are depicted.

With respect to frame 1000 (first frame), the following scene is shown:Forwarder A is running towards goal and making a shot in direction ofteammate B. The probabilities are as follows: At the current moment theprobability of teammate B for receiving the ball is at 30% and for shoton goal at 10% (based on: B position close to opponents goal, positionof ball, A and C.

With respect to frame 1002 (second frame), the following scene is shown:Teammate B is receiving the ball from A. The probabilities are asfollows: At the current moment the probability of teammate B forreceiving the ball is at 100% (based on: just receives the ball) and forshot on goal at 50% (based on: Position close to opponents goal,position of C).

With respect to frame 1003, the following scene is shown: Teammate B isexecuting a shot towards the goal. The probabilities are as follows: Atthe current moment the probability of teammate B for receiving the ballis at 0% and for shot on goal at 100% (based on: Just executes a shot ongoal).

Referring to FIG. 11, three frames 1100, 1101, 1102 are depicted.

With respect to frame 11.00 (first frame), the following scene is shown:Forwarder A is running towards goal and making a shot in direction ofteammate B. The probabilities are as follows: At the current moment theprobability of teammate B for receiving the ball is at 30% and for shoton goal at 10% (based on: B position close to opponents goal, positionof ball A, and C).

With respect to frame 1102 (second frame), the following scene is shown:Suddenly goalkeeper C is going to jump towards the ball. Theprobabilities are as follows: At the current moment the probability ofteammate B for receiving the ball is at 50% (based on: just before thecatching) and for shot on goal at 20% (based on: Position close toopponents goal, position of C).

With respect to frame 1103 (third frame), the following scene is shown:Teammate C catched the ball and is shooting it out of the danger zone.The probabilities are as follows: At the current moment the probabilityof teammate B for receiving the ball is at 0% and for shot on goal at 0%(based on: opponent Chas ball).

The probability tree provides information to lookup the animationdatabase to search for possible animations for the player. Logical,technical attributes and patterns are explained above in AnimationDatabase. These are attributes of the stored animations, whereas fixedand dynamic attributes as explained in Early Event Detection areattributes of the probability tree and are bound to the actual playerand scene.

One position is provided for each player at each time (e.g. 25 times persecond). At least it is needed to define a pattern and to perform alookup.

Additional positions would be used for distinctive body parts as feet orarms, these are used to match in more details to the live scene. If leftfoot is in front it would be inserted like that in the animation queue.

A probability animation list is generated. The target is to find theanimations which could be used for the current situation—FIG. 12illustrates this principle. After starting the process in step 1201 theprobability tree is received (step 1202). The use of the information“Has Ball Possession” from the probability tree reduces the number ofpossible animations our example from 3000 (all animation, step 1203) to1000 (animations in category ‘With Ball’, step 1204) as e.g. ones areexcluded where a player stands or jumps for a header. As the exemplaryprobability tree indicates to lookup a shot 200 possible animations areleft (1204). Further observing logical attributes, e.g. the playershoots with the tight foot to the left using a sidefoot shot (seen fromhis orientation, step 1205), which gives 12 animations left. Technicalattributes as the current running speed of the player and its shotstrength will further reduce the amount of possible animations to 4animations in the end (step 1206). The resulting probability animationlist 1208 includes a probability for each of these four animations 1207.This probability can further be enhanced by observing movement patternson the positional data (and ball). The more positional and furtherinformation are received for a single player at each time the better thepattern recognition is.

Whereas the workflow for objects is based on physics as described abovethe reconstruction of characters is different utilizing variousanimation techniques. As an advantage of the workflow the reconstructionwill always look realistic, but varies from the real motion due to thepre-recorded motion capture animation and physical simulation. To solvethis, the animation will be placed ideal in the animation queue—e.g. ina way that the feet position matches and also during the optimizationstep afterwards important details will be adjusted to match the realmotion better.

FIG. 7 illustrates the general method for the reconstruction ofcharacters. Multiple stages (layers) are used for organizationalpurposes; these can differ for other disciplines of sports or usage.FIG. 8 shows the different stages, layers and the usage of weights.

The early event detection results in a Probability Tree (FIG. 9) andfinally in a Probability Animation List (FIG. 12). This information isused to generate the following three stages of the animation data queue.

In a stage 1 a basic animation is provided (step 809). It is a basiclayer of the animation queue which is always generated, so overlaidanimation can be blended in and out at any time. Event correction, if anEvent of the Early Event Detection has been detected as wrong afterwardsan event correction is done. This means already inserted animations intothe queue might be cancelled and another animation be inserted based onthe results.

In a stage 2 action animation and/or physical behaviour (step 812) maybe used for amendment or correction of the basic animation. For specificactions, e.g. shot or tackling, this stage may be used. Action animationworks similar to the basic animation with motioncapture animations.Physical behaviour is in some cases used in combination or instead.Animations of stage 2 may have a higher priority, so overwrite animationdata sets of stage 1, for the case that an action animation is playedwith a weight of 100%.

In a stage 3 an optimization (step 814) of the animation data sets maybe done. The focus may be to get closer again to reality, to adjust theconvincing realistic animation in a way that it stays convincing, butdetails are adjusted to match reality.

The intention of the first stage is to provide a basic animation whichcan always be used or can always be blended into from action animationor physical behaviour. So it is always part of the animation queue, evenif its weight is 0 (e.g. actually a shot (from stage 2) overlays thebasic animation). In such case when the basic animation is completelyblended out it can be adjusted to match the overlay animation, so thatat any time a blending into a running animation can be proceeded.Besides this it does have its own probability tree, performs a differentanimation database lookup than the action animation and therefor resultsin its own probability list.

Animations used in this stage include basic movement as idle, walk,trot, running, running sideward and the like—also including animationsfor ball control like dribbling but not any actions as shooting,tackling etc. Most basic animations are loopable, but there areexceptions (e.g. making a sudden side step). The probability tree isdifferent to the action animations and e.g. doesn't handle or has adetection for a shot as the action animation's tree like in FIG. 9. Thenumber of animations is much smaller than for the action animation soalso the animation database lookup is reduced resulting in a smallerprobability animation list.

When further positions are received the system will consider these, e.g.positions for the feet will make sure these are matching in theresulting animation.

The animations can be split into upper and lower body, e.g. thereforejust a lower body animation for running could be used whereas the upperbody is still doing an action animation. In addition the blending can bevaried for both. E.g, for a case that in both animations the lower bodymovements are matching well, the lower body could be blended in a shorttime into the other animation and the upper body could be blended slowlywhen its movements are very different.

Within the predicted time range a pre-recognition will use detectedevent assumptions to process a lookup in the animation database andinsert the best matching animation into the (future) animation queue. Asit only a prediction further event details can come in some timelater—the animation will be adjusted, quickly blended into another orreplaced by another one,

Action animations may overlay the basic animation, but will only beactive when an action animation should be played as described in FIG. 8and FIG. 13 (Stage 2). If the moment of the player shooting a ballarrives (step 815), the shot is detected by the early event detectionwhich refreshes the probability tree (exemplary illustrated in FIG. 9).A specific shot action animation will be delivered by the animationdatabase lookup as in FIG. 12. The animation with the highest prioritywill be inserted into the animation queue and its weight will increasewhen either the important part of the animation comes closer (moment ofshot) or the probability (in probability animation list) rises above allother options. Furthermore the event correction detects possible errorsin this stage and solves these.

The insertion of an action animation is illustrated in FIG. 13. Thebasic animation in animation queue is updated if an action animation orphysical behaviour will be inserted, to match it perfectly (end ofanimation, for a possible blending over to the basic animation).Therefor the basic animation is split and with using locomotion allowsperfectly blending from basic animation into an action animation andvice versa.

In combination with the action animation or instead of it the physicalbehaviour animation can be used. Filling in missing or complex animationwhere physical behaviours play a role and there is no matching animation(e.g. collision of players) further described above with respect to theaspect of physical simulation.

Revise and optimize animations from the animation queue may be donebased on positional data. Match orientation of head, eyes and exactpositions of elbow, hand, fingers etc. Depending on what positions aredelivered by the tracking system. As illustrated in FIG. 13 it can beused to optimize a player shooting a ball. Where the direction of thelive scene's ball trajectory is different to the one recorded withmotion-capture. With blending in inverse kinematic over the underlaidanimation and just for the shooting foot the process of a player kickingthe ball will be adjusted so it looks more natural according to the balltrajectory. Same applies to optimization of arms and hands for the goalkeeper. Another example is the orientation of the character's head, inmost cases in football the player watches towards the ball, so that thehead can be orientated towards the ball.

following, further possible applications for the method of 3Dreconstruction are described.

Live animation may be performed. What makes the technology unique, amongother things, is that it is possible (with the method of livereconstruction) to change the “live” situations and interact with these.The generated 3D animation is interactive for users—these can operatewith the camera, insert analytical elements and change the virtualenvironment in other ways. FIG. 15 shows the concept of watching thefootball event on a television screen and at the same time interactingwith the 3D reconstruction on a tablet which displays the live scene insync. This is the so called “Second Screen Experience”.

In a way this is basically close to a standard broadcast of a match butextends the way to watch scenes by using camera angles not possible innormal broadcast (place it behind players on the field). There can beautomatic cameras and players can be selected by the consumers which thecamera is focusing on. Extra information like current condition of aplayer and statistics can be displayed to make the live experienceentertaining. Predicting actions of players or evaluating andvisualizing probabilities. Other views are also possible, as the birdview as shown in FIG. 16.

Target customers are broadcasters, news/media agencies, web portals,club websites as well as the direct sport enthusiast accessible ontarget devices in interactive web I smartphone I tablet applications.Interactivity gives the user full freedom to watch the scene from everyperspective, enabling analytical elements and in a next step the usercould also have the option to generate his own view on the goal, renderit as a video and share it with friends.

In a virtual studio environment the live reconstruction can be used todisplay small holographic players on a moderators table. Technically anaugmented reality product—the tactic table allows experts to analyze thescene live during the game. This puts the expert or commenter in front,and his view onto the live scenery. It could be very interesting toresearch the opportunities to blend from full 3D to the augmentedreality representation or vice versa. This would enhance the product ina way to not rely on the studio camera but also to blend over to full 3Dand have the full range of camera angles.

A multiplatform-solution may be provided. FIG. 14 illustrates theconcept of a centralized server system to generate reconstructions fordifferent target devices. The method of the reconstruction is executedon the server and the results are made available for a playback softwareon the devices. Different asset libraries (animation database isdifferent) are used to support different quality levels of hardwaredevices. So a high-end workstation can be used as well as a smartphone.

The system depicted in FIG. 14 is provided as centralized server system.By a tracking system 1400 3D positional data for the live scenes areprovided. The detected data may be stored in a storage device 1401 whichis connected to a central server device 1402. According to FIG. 14, thecentralized server device 1402 is connected to a library device 1403, aserver storage 1404, and another library device 1405 providing a smallanimation library (Low-Detail Models). The central server device 1402may provide the virtual 3D reconstruction animation to different displaydevices 1406, 1407 which, according to the embodiment in FIG. 14, areprovided by a high-end engine 1406 and a multi-platform engine 1407. Thehigh-end engine 1406 may be a broadcast device for rap video or stream.The multi-platform engine 1407 may refer to mobile devices such asmobile phones or tablet computers using different operating systems.

What is claimed is:
 1. A method for real-time virtual 3D reconstructionof a live event broadcast including a plurality of participants, thevirtual 3D reconstruction utilizing an animation system including aprocessor, a database, an input device, and an output device, the methodcomprising: determining a first live action by a participant and liveactions by other participants of the plurality of participants in thelive event broadcast by analyzing 3D positional tracking data for theplurality of participants in the live event broadcast; determining anaction probability for future live actions that the participant may makeafter the first live action by comparing the first live action by theparticipant to the live actions by the other participants; generating anaction probability tree of virtual actions corresponding to the futurelive actions that may be made by the participant after the first liveaction by the participant in the live event broadcast, the actionprobability tree including a basic animation of the highest probabilityaction for the future live actions, the basic animation being a basiclayer of a virtual action permitting an overlaid animation to be blendedin and out of the basic animation; selecting a first virtual action fromthe action probability tree with the highest probability; generating a3D animation data set including the basic animation for the firstvirtual action; modifying the 3D animation data set with an actionanimation to correct or amend the basic animation for the first virtualaction, the action animation being selected to correspond to the basicanimation and including further animation details of the first virtualaction corresponding to the future live action predicted to be made bythe participant in the live event broadcast; and providing the 3Danimation data set of the first virtual action to he output device todisplay the virtual 3D reconstruction of the live event broadcast. 2.The method of claim 1, further comprising optimizing the modified 3Danimation data set to include additional animation details to the firstvirtual action to further correspond to the future live action predictedto be made by the participant in the live event broadcast.
 3. The methodof claim 1, further comprising: comparing the first virtual action witha second live action, the second live action being made by theparticipant and being after the first live action; and if the firstvirtual action does not match with the second live action, selecting asecond virtual action from the action probability tree with the nexthighest probability; generating a 3D animation data set including thebasic animation for the second virtual action; and providing the 3Danimation data set of the second virtual action to the output device todisplay the virtual 3D reconstruction of the live event broadcast. 4.The method of claim 1, further comprising generating an animation dataqueue by repeating one or more of: the determining of the first liveaction by the participant; the generating of the 3D animation data set;and the providing of the 3D animation data set.
 5. The method of claim4, further comprising performing an event correction for the animationdata queue by at least one of: deleting one of the plurality of 3Danimation data sets from the animation data queue; and modifying one ofthe plurality of 3D animation data sets from the animation data queuewhen a new circumstance occurs in the live event broadcast.
 6. Themethod of claim 5, wherein the at least one of the deleting and themodifying of the one of the plurality of 3D animation data sets includessubstituting a 3D animation data set selected from the animation dataqueue with a new 3D animation data set determined to closer animate thefirst virtual action.
 7. The method of claim 4, wherein the generatingof the animation data queue occurs in a delayed time range, the delayedtime range being between a live time and a playback time of the liveevent broadcast, the playback time being the current time of playback ofthe animation data queue, further comprising determining a second liveaction made by the participant after the first live action during thedelayed time range, the second live action having a higher influence onthe action probability for future live actions than other actionprobabilities in the action probability tree.
 8. The method of claim 5,wherein the event correction is based on the action probability for thefuture live actions, animations in the animation data queue, and the newcircumstance in the live event broadcast, the performing the eventcorrection occurring in a delayed time range, the delayed time rangebeing between a live time and a playback time of the live eventbroadcast, the playback time being the current time of playback of theanimation data queue.
 9. The method of claim 1, wherein the 3D animationdata set is provided in different data formats, the data formats beingadjusted for different display devices.
 10. The method of claim 1,wherein the plurality of 3D animation data sets include simulated 3Danimation data and pre-recorded motion capture data.
 11. An animationsystem for real-time virtual 3D reconstruction of a live event broadcastincluding a plurality of participants, the animation system comprising:a database; an input device; an output device; and a processorconfigured to: determine a first live action by a participant and liveactions by other participants of the plurality of participants in thelive event broadcast by analyzing 3D positional tracking data for theplurality of participants in the live event broadcast; determine anaction probability for future live actions that the participant may makeafter the first live action by comparing the first live action by theparticipant to the live actions by the other participants; generate anaction probability tree of virtual actions corresponding to the futurelive actions that may be made by the participant after the first liveaction by the participant in the live event broadcast, the actionprobability tree including a basic animation of the highest probabilityaction for the future live actions, the basic animation being a basiclayer of a virtual action permitting an overlaid animation to be blendedin and out of the basic animation; select a first virtual action fromthe action probability tree with the highest probability; generate a 3Danimation data set including the basic animation for the first virtualaction; modify the 3D animation data set with an action animation tocorrect or amend the basic animation for the first virtual action, theaction animation being selected to correspond to the basic animation andincluding further animation details of the first virtual actioncorresponding to the future live action predicted to be made by theparticipant in the live event broadcast; and provide the 3D animationdata set of the first virtual action to the output device to display thevirtual 3D reconstruction of the live event broadcast.
 12. The animationsystem of claim 11, wherein the processor is further configured tooptimize the modified 3D animation data set to include additionalanimation details to the first virtual action to further correspond tothe future live action predicted to be made by the participant in thelive event broadcast.
 13. The animation system of claim 11, wherein theprocessor is further configured to: compare the first virtual actionwith a second live action, the second live action being made by theparticipant and being after the first live action; and if the firstvirtual action does not match with the second live action, select asecond virtual action from the action probability tree with the nexthighest probability; generate a 3D animation data set including thebasic animation for the second virtual action; and provide the 3Danimation data set of the second virtual action to the output device todisplay the virtual 3D reconstruction of the live event broadcast. 14.The animation system of claim 11, wherein the processor is furtherconfigured to generate an animation data queue by repeating one or moreof: the determination of the first live action by the participant; thegeneration of the 3D animation data set; and the provide of the 3Danimation data set.
 15. The animation system of claim 13, wherein theprocessor is further configured to perform an event correction for theanimation data queue by at least one of: deleting one of the pluralityof 3D animation data sets from the animation data queue; and modifyingone of the plurality of 3D animation data sets from the animation dataqueue when a new circumstance occurs in the live event broadcast. 16.The animation system of claim 14, wherein the at least one of thedeleting and the modifying of the one of the plurality of 3D animationdata sets includes substituting a 3D animation data set selected fromthe animation data queue with a new 3D animation data set determined tocloser animate the first virtual action.
 17. The animation system ofclaim 13, wherein the generation of the animation data queue occurs in adelayed time range, the delayed time range being between a live time anda playback time of the live event broadcast, the playback time being thecurrent time of playback of the animation data queue, wherein theprocessor is further configured to determine a second live action madeby the participant after the first live action during the delayed timerange, the second live action having a higher influence on the actionprobability for future live actions than other action probabilities inthe action probability tree.
 18. A non-transitory computer-readablemedia including a computer program for performing a process of runningan animation system for real-time virtual 3D reconstruction of a liveevent broadcast, the animation system including a processor, a database,an input device, and an output device, the process comprising:determining a first live action by a participant and live actions byother participants of the plurality of participants in the live eventbroadcast by analyzing 3D positional tracking data for the plurality ofparticipants in the live event broadcast; determining an actionprobability for future live actions that the participant may make afterthe first live action by comparing the first live action by theparticipant to the live actions by the other participants; generating anaction probability tree of virtual actions corresponding to the futurelive actions that may be made by the participant after the first liveaction by the participant in the live event broadcast, the actionprobability tree including a basic animation of the highest probabilityaction for the future live actions, the basic animation being a basiclayer of a virtual action permitting an overlaid animation to be blendedin and out of the basic animation; selecting a first virtual action fromthe action probability tree with the highest probability; generating a3D animation data set including the basic animation for the firstvirtual action; modifying the 3D animation data set with an actionanimation to correct or amend the basic animation for the first virtualaction, the action animation being selected to correspond to the basicanimation and including further animation details of the first virtualaction corresponding to the future live action predicted to be made bythe participant in the live event broadcast; and providing the 3Danimation data set of the first virtual action to he output device todisplay the virtual 3D reconstruction of the live event broadcast. 19.The non-transitory computer-readable media of claim 18, the processfurther comprising optimizing the modified 3D animation data set toinclude additional animation details to the first virtual action tofurther correspond to the future live action predicted to be made by theparticipant in the live event broadcast.
 20. The non-transitorycomputer-readable media of claim 18, the process further comprising:comparing the first virtual action with a second live action, the secondlive action being made by the participant and being after the first liveaction; and if the first virtual action does not match with the secondlive action, selecting a second virtual action from the actionprobability tree with the next highest probability; generating a 3Danimation data set including the basic animation for the second virtualaction; and providing the 3D animation data set of the second virtualaction to the output device to display the virtual 3D reconstruction ofthe live event broadcast.